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(57) Abstract 

During an actual medical procedure, 
the actions of a medical instrument (16) 
are measured and recorded. These actions 
are generated on a member representative 
of the medical instrument to simulate the 
actual medical procedure. Forces acting 
upon the medical instrument (16) are 
sensed and simulated in the member using 
a force/tactile reflecting mechanism. The 
preferred embodiment uses virtual reality 
technology including image processing, 
three-dimensional graphics and display 
methods, simulated force/tactile reflection, 
head/hand movement, position sensing, 
and sound generation to provide an 
accurate simulation of endoscopic medical 
procedures. 
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METHOD AND SYSTEM FOR 
SIMULATING MEDICAL PROCEDURES 
INCLUDING VIRTUAL REALITY AND CONTROL 
METHOD AND SYSTEM FOR USE THEREIN 

5 Cross-Reference To Related App lication 

This application is a continuation-in-part of 
copending U.S. application Serial No. 984,324, filed 
December 2, 1992. 

Technical Field 

10 This invention relates to methods and systems 

for simulating medical procedures for training and for 
remote control. 

Background Art 

Laparoscopic surgery, a specific application 
15 of endoscopic surgery, is performed by: 

(1) making several small openings into the 
peritoneal cavity 10 of the body 12 as shown in Figure 
i; 

(2) insufflating with an inert gas, C0 2 
20 (providing medium for adequate anatomical inspection) ; 

(3) inserting a viewing scope 14 (laparo- 
scope) into one opening 10; 

(4) inserting othe;. 3urgical instruments 16 
into the other openings (i.e. rapiers, cutting/cauter- 

25 izing instruments, etc.); and 

(5) maneuvering and manipulating these 
instruments 16 on the basis of tactile feel and the 
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visual information of the surgical site from the insert- 
ed viewing scope 14 (usually displayed on a TV monitor) . 

Because the feel of the instruments 16 and the 
imagery of the surgical site is restricted to a small 
5 local area which is not directly visible, surgeons must 
train extensively (usually on animals) prior to perform- 
ing human procedures. This type of surgery is still 
very much an art rather than science. 

Endoscopic and laparoscopic intervention is 
10 being fervently applied to procedures (such as cholecys- 
tectomies, hernia repair, nephrectomies, appendectomies, 
and numerous gynecological procedures) being praised for 
fast patient recovery time and decreased hospital and 
surgery costs. As new applications emerge, training 
15 will play a crucial role in technological adoption. A 
need exists for a training device which will facilitate 
the transition from more traditional open procedures to 
endoscopic procedures by fostering the surgical skill 
necessary for the execution of efficient and complica- 
20 tion-free procedures. 

U.S. Patent No. 5,149,270, issued to McKeown, 
discloses an apparatus for practicing endoscopic surgi- 
cal procedures. The apparatus has a cavity in which an 
object simulating a human organ is mounted for perform- 

25 ing a practice procedure. The cavity is closeable to 
outside view, thereby forcing the individual practicing 
the procedure to use and manipulate the instruments 
relying only upon a viewing scope in an attempt to mimic 
the actual diagnostic and operating conditions. This 

30 configuration, by relying upon physical linkages and 
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organ materials, lacks the versatility, robustness, and 
repeatability of a programmable simulation device. 

International PCT application No. 
PCT/US92/02186, published 1 October 1992, discloses an 
5 automated surgical system and apparatus. An endoscopic 
instrument and a viewing scope are controlled remotely 
via a control mechanism in communication with the 
surgical instruments over a telecommunications link. 
This device, while directed toward the remote control of 

10 surgical instruments, does not address the necessity of 
force feedback for giving the remote user the feel of 
actual surgical procedures. Further, the device is 
directed toward remotely performing actual surgical 
procedures rather than simulated surgical procedures for 

15 the purposes of training. 

Summary of the Invention 

It is an object of the above invention to 
provide a method and system for simulating actual 
medical procedures. 

20 A further object of the above invention is to 

provide a method for providing a virtual reality in 
response to a position and orientation of a member 
representative of a medical instrument. 

Moreover, an object of the above invention is 
25 to provide a method and system for controlling a medical 
instrument . 



'■ 
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An additional object of the above invention is 
to provide a four-axis controller for a medical instru- 
ment . 

In carrying out the above objects, the present 
5 invention provides a method for simulating actual 
medical procedures which utilize at least one medical 
instrument. According to the present invention, the 
actions of the medical instrument are measured during an 
actual medical procedure. These measured actions of the 
10 medical instrument are generated on a member represen- 
tative of the medical instrument so as to simulate the 
actual medical procedure. 

In carrying out the above objects, the present 
invention further provides a system for simulating 

15 actual medical procedures which utilize at least one 
medical instrument . The system comprises means for 
measuring the actions of the medical instrument during 
an actual medical procedure. Means are also provided 
for generating the measured actions on a member repre- 

20 sentative of the medical instrument whereby the actions 
of the member simulate the actual medical procedure. 

In carrying out the above objects, the present 
invention also provides a method for providing a virtual 
reality in response to a position and orientation of a 

25 member representative of a medical instrument. An 
electrical signal is generated for each of a plurality 
of degrees of freedom of the member as a function of the 
position and orientation of the member in three-dimen- 
sional space. At least one virtual reality force field 

30 is generated in response to the generated signals. A 
force signal is generated for each degree of freedom as 



WO 95/02233 



PCT/US94/07218 



a function of the force field. A force on the member is 
generated for each force signal, thus providing the 
virtual reality. 

In carrying out the above objects, the present 
5 invention additionally provides a method for controlling 
a medical instrument. The position and orientation of 
a member representative of the medical instrument is 
measured. Actions are generated on the medical instru- 
ment based upon the measured position and orientation of 
10 the member. The forces acting upon the medical instru- 
ment are measured. Forces are generated on the member 
based upon the measured forces acting upon the medical 
instrument . 

In carrying out the above objects, the present 
15 invention also provides a system for controlling a 
medical instrument. The system comprises means for 
measuring the position and orientation of a member 
representative of the medical instrument. Means are 
provided to generate actions on the medical instrument 
20 based upon the measured position and orientation of the 
member. The system further comprises means for measur- 
ing the forces acting upon the medical instrument. 
Means are provided to generate forces on the member 
based upon the measured forces acting upon the medical 
25 instrument. 

In carrying out the above objects, the present 
invention also provides a four-axis controller for a 
medical instrument. The system comprises a member 
representative of the medical instrument. A four-axis 
30 interface device, which is responsive to the position 
and orientation of the member, is secured to the member. 
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Electrical signals representative of the position and 
orientation of the member are generated in the four-axis 
interface device. Means for actuating the medical 
instrument in response to the electrical signals are 
5 included. Means for generating a force field based upon 
a force induced upon the medical instrument are provid- 
ed. A force signal for each axis of the interface 
device is generated by further means, with the actuators 
responsive to these force signals to provide a force to 
10 each axis of the interface device for reflecting a force 
on the medical instrument to the member. 

Brief Description Of The Drawings 

FIGURE 1 shows the insertion of a scope and an 
instrument into a body during a laparoscopic medical 
15 procedure; 

FIGURE 2 is a block diagram representation of 
the recording apparatus; 

FIGURE 3 depicts the systematic scanning of an 
anatomical feature to generate a complete imagery 
20 record; 

FIGURE 4 is a block diagram representation of 
the virtual reality training/replay apparatus; 

FIGURE 5 is a block diagram representation of 
the incorporation of the recording apparatus with the 
25 virtual reality training/replay apparatus; 
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FIGURE 6 is a block diagram representation of 
using the virtual reality replay system to provide a 
surgeon with preoperative imagery; 

FIGURE 7 is a block diagram representation of 
5 using the virtual reality system in conjunction with 
robotic endoscopic medical equipment; 

FIGURE 8 is a block diagram representation of 
the force/tactile subsystem control concept; 

■ 

FIGURE 9 is a schematic representation of the 
10 laparoscopic simulator; and 

FIGURE 10 shows the four-axis force reflecting 
module within an opening in the laparoscopic simulator. 

Best Mode For Carrying Out The Invention 

Described is a method and an apparatus which 
15 uses Virtual Reality Technology including image process- 
ing, three dimensional graphics and display methods, 
simulated force/tactile reflection, programmed stereo- 
phonic sound generation, head/hand movement and position 
sensing to provide an accurate control or simulation of 
20 medical procedures, specifically: endoscopic medical 
procedures including, but not limited to, cholycystec- 
tomies, vasectomies, bowel resections, inguinal and 
bilateral hernia repairs, appendectomies, lymphadenecto- 
mies, vagotomies, bronchial surgeries, hysterectomies, 
25 nephrectomies, lysis of adhesions, tumor extraction, 
knee arthroscopies, and abdominal explorations; micro- 
surgery, i.e. viteoretinal, otologic, laryngeal and 
stereotactic procedures; intubations, including neonatal 
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procedures; and arterial and venous replacement and 
bypass methods used, for example, in treatment of 
innominate and subclavian occlusive disease. 

The apparatus can be used to provide realistic 
5 simulations of patients on which to practice medical 
procedures for training and qualification purposes, to 
provide a means for trainees to see and feel actual (or 
simulated) surgeries, to display information from 
previously obtained medical diagnostics or image modali- 
10 ties (CT data, PET data, MRI data, etc) to the surgeon 
overlaid on current image information during the sur- 
gery, and to provide the man-machine interface for 
robotic surgery. 

The apparatus consists of a two or three 
15 dimensional display device, a two or three dimensional 
sound device, a graphics /image processing engine and 
storage module capable of real-time medical image 
generation, and programmable tactile/force reflecting 
mechanisms which can generate the "feel" of medical 
20 instruments and the interaction of these instruments 
with the anatomical simulation. The preferred embodi- 
ment of the invention is designed for simulating endo- 
scopic medical procedures due to the minimally invasive 
nature of these techniques. 

25 The preferred embodiment of this invention 

addresses the problems presented by the prior art by 
providing means for controlled, preprogrammable simula- 
tion of a medical procedure (using both active tactile 
feel and imagery) so that standardized practice and 

30 qualification sessions can be performed by surgeons. 
The same embodiment can also be used to augment the data 
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normally delivered to the surgeon in actual endoscopic 
surgery, and can provide surrogate feels and imagery- to 
surgeon trainees from actual surgeries being performed 
by expert surgeons. 

5 The basic medical procedure simulation system 

consists of two basic functions: 

(1) measuring and recording sights, sounds, 
and feels generated during actual surgery; 

(2) accurately playing back these recorded 
10 sights, sounds, and feels and for using the recorded 

data to generate new information necessary to emulate 
the responses to alternative actions taken by a surgeon 
trainee . 

Turning now to Figure 2, the recording appara- 

15 tus 20 that allows the critical events in a medical 
procedure to be measured and recorded during actual 
surgeries performed by expert surgeons is shown. 
Medical instruments 16 having pressure and position 
sensors 22 are connected to a data digitizer 24 . The 

20 pressure and position data from the digitizer 24 is 
recorded by a recording system 26 consisting of a 
computer and a storage subsystem. A viewing scope 14 
having pressure and position sensors 28 is connected to 
an image digitizer/recorder 30. The image digitiz- 

25 er/recorder 30 is also connected to the recording system 
26. The recording system 26 can either compress and 
store digital image data or store analog video on a 
computer- controlled video cassette recorder (VCR). 
The non- image data is stored on digital disk or tape 

30 storage. An accurate time base 32 is also connected to 
the recording subsystem 26 to provide time stamps for 
the instrument locations, forces or pressures, and 
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actions such as trigger actuations during the specified 
scope locations, forces (or pressures) , and image data 
captured during the operation. Each data item (force, 
position, trigger action item or image captured) is time 
5 stamped so that it is placed accurately into the time 
sequence of events during surgery. The recording 
apparatus 20 in Figure 2 can be directly connected to a 
playback apparatus (in Figure 5) to provide immediate 
feedback to a trainee, 

10 To support the generation of data and images 

to accommodate various user inputs, which will be 
different than the mechanics implemented in the operat- 
ing room and will differ from any particular surgery 
performed, a systematic scanning of anatomical features 

15 is performed. Figure 3 depicts the systematic scanning 
of anatomical features using the scope 14 to generate a 
complete imagery record of organ and tissue orientation 
surrounding the site ' of medical simulation. This 
supports generation of in-between data and interpolated 

20 views based on recorded data. 

The virtual reality training/replay apparatus 
40 needed to playback the recorded data is incorporated 
in Figure 4. This apparatus 40 generates the virtual 
space which displays the time stamped and interpolated 

25 data stored in the image and data storage system 42. A 
virtual reality control computer 44 receives image and 
non- image data from the image and data storage system 
42. Position, orientation, and force data are processed 
by the computer 44 and sent to the force/tactile pro- 

30 grammed controller and hand/instrument tracking subsys- 
tem 46. The force/tactile programmed controller and 
hand/ instrument tracking subsystem 46 is connected to a 
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tactile/force reflecting subject simulation mechanism 48 
in order to provide force feedback. 

The virtual reality control computer 44 
obtains image data from the image and data storage 
5 system 42 based on the position and orientation of the 
instruments. The control computer 44 processes this 
data and relays it to an image warp processor 50. The 
image warping function 50 uses morphing, defined as the 
correspondence of two images through corresponding 

10 polygon patches, as the means for providing interpolated 
images from data collected during recording sessions. 
A 3D graphics generator 52 is also connected to the 
virtual reality control computer 44 in order to generate 
accurate three dimensional images of endoscopic instru- 

15 ments. A graphics overlay unit 54 is used to overlay 
the three dimensional images from the 3D graphics 
generator 52 on the morphed video derived imagery from 
the image warp processor 50 . The output of the graphics 
overlay unit 54 is fed into a three dimensional display 

20 system 56. 

An optional (denoted by the dashed lines) 
trainee head tracking system 58 can be connected to the 
virtual reality control computer 44 to provide informa- 
tion for the three dimensional display generation 

25 function to control generation of view dependent imagery 
assuming that the display system 56 is a head mounted 
system. Alternative embodiments for the display system 
56 which use more conventional display technology, such 
as CRTs, might not require head tracking systems 58. 

30 However, future systems are expected to replace CRT 
viewing of endoscope -derived image data with head- 
mounted three-dimensional viewing. This viewing ar- 
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rangement will require the head position sensing subsys- 
tem 58. 

The inclusion of a head position sensing 
subsystem 58 also allows the incorporation of a three- 
5 dimensional sound generation system 60, The sound 
generation system receives audio information from the 
virtual reality control computer 44 and sends this 
information to a sound system 62. The use of sound 
generation is also optional since it is not necessary in 
10 simulating endoscopic procedures. 

Figure 5 shows how the recording apparatus 20 
in Figure 2 can be used to acquire image and manipula- 
tion data 70 which is then used to compose simulation 
data sets 72 for replay on the virtual reality replay 

15 apparatus 40 from Figure 4. An experienced surgeon 74 
performs an actual endoscopic medical procedure using 
endoscopic instruments 76. The recording system 20 is 
used to measure and record the sights, sounds, feels, 
and actions occurring during the actual medical proce- 

20 dure. The sights, sounds, feels, and actions data are 
stored in the measured data and image storage system 70. 
Simulation data sets 72 are formed off-line from the 
measured data 70 by a data composition system 78. The 
virtual reality training system 40 uses images and data 

25 in the simulation database 72 to allow a trainee 80 to 
explore this data. The trainee 80 can view, hear, and 
feel operations through manipulation and positioning of 
the simulated instruments, or based on his head loca- 
tion. Images or data not in the simulation database 72 

30 are generated through interpolation between images/data 
items which are in the . database 72 (interpolation 
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between images is accomplished through morphing 
mapping predetermined image polygons between images) . 

Figure 5 also shows how the recording appara- 
tus 20 can be used to pick-up data and imagery from an 
5 in-progress medical procedure for immediate replay on 
the virtual reality replay/training apparatus 40. In 
this mode, the virtual reality replay apparatus 40 
immediately receives the output of the recording appara- 
tus 20 via a communications channel 82. Therefore, the 

10 trainee 80 does not explore the input data and imagery 
through manipulation, but rather experiences the data 
stream. This data includes the virtual forces generated 
by the force/tactile subsystem and applied to the 
trainee's hands through the simulated instruments. 

15 Thus, the trainee 80 can gain intuitive understanding of 
the medical procedure, including the feelings, sights, 
and sounds of the operation, without the risks and 
possible complications associated with medical inter- 
vention. 

20 Figure 6 shows how elements of the virtual 

reality replay system 40 can be used to provide the 
surgeon 90 with views of preoperative information and 
imagery 92, such as CT scans and MRI scans, to augment 
what is available now for endoscopic procedures (for 

25 instance, video imagery from the laparoscope) . A 
surgeon 90 performs an actual medical procedure using 
endoscopic instruments 94. The recording system appara- 
tus 20 is used to measure and record the actions and 
images in the instruments 94 during the procedure. The 

30 virtual reality training system 40 incorporates pre- 
operative test data and images 92 with the output of the 
recording apparatus 20. The use of virtual reality 
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display techniques allows metabolic and anatomical 
structure graphics to be overlayed on the basic video 
imagery now available to the surgeon 90 this is 
especially valuable if we provide the surgeon 90 with a 
5 three dimensional viewer (such as a head-mount three- 
dimensional viewer) for graphically enhanced anatomical 
information. 

Figure 7 shows how the virtual reality systems 
described already can also be used to provide feedback 

10 from and control information to robotic endoscopic 
medical equipment 100. A remote surgeon's 102 instru- 
ments, viewing scope, and head position are measured and 
recorded by the virtual reality training system 104 . 
This data is transmitted to the robotic endoscopic 

15 surgery equipment 100 via a communications channel 106. 
A local surgeon's 108 instrument, scope, and head 
position is also measured and recorded by a second 
virtual reality training system 110. The robotic 
instruments are controlled to perform the actions of 

20 either the remote surgeon 102 or the local surgeon 108 
on the patient 112. • The resulting sights, sounds, 
actions, and feels are experienced by both the remote 
surgeon 102 and the local surgeon 108. The virtual 
reality concept described in this document provides the 

25 means for sharing control of such robotic equipment 100 
between several surgeons or trainees. 

The most important subsystems in the virtual 
reality replay apparatus are the force/ tactile control- 
ler 46 and the tactile/force reflecting mechanism 48. 
30 These systems use force reflection information generated 
from the simulation database (or model-base) to create 
the feel of the medical instruments. Figure 8 shows a 
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control concept for creating the feel of medical instru- 
ments during a simulation. The hands 120 of a surgeon 
or trainee grasp and move a member representative of an 
instrument. The position of the instrument 122 is 
5 determined and fed into a geometric model of an organ 
124. The geometric model 124 includes data representing 
the organ size, position, surface characteristics (in 
terms of elasticity, resistance to searing, slipperi- 
ness, etc.). Forces and torques are applied to the 

10 surgeon's/trainee's hands 120 from a force/torque 
subsystem 126 using data from the geometric model 124. 
Thus, realistic feels are produced on the simulated 
medical instrument representing instrument-to-organ 
interaction forces based on the position of the instru- 

15 ment . 

An apparatus for simulating laparoscopic 
medical procedures shown in Figure 9 uses the servo 
systems concept described in Figure 8 to the provide 
force- feedback control. The apparatus consists of a 

20 box-like enclosure 130 which provides three openings 132 
for insertion of instruments 134. When the instruments 
134 are inserted they are clamped to four-axis devices 
136 such as the force reflection mechanism 140 shown in 
Figure 10. This mechanism 140 includes, in effect, a 

25 small backdrivable robot which moves in 4 degrees of 
freedom: 1 linear position (extension) and 3 attitudes 
(roll, pitch, and yaw) . This is accomplished by the 
extension actuator 142, roll actuator 144, pitch actua- 
tor 146, and yaw actuator 148. This mechanism 140 can 

30 translate and provide forces/ torques under programmed 
control (generated by the modeling subsystem in Figure 
8) which feel to the trainee like the interactions 
characteristic of the actual medical procedure. These 
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four-axis actuation systems are reduced implementations 
of universal force reflection mechanisms described in 
co-pending application Serial No. 984,324, filed Decern- 
- ber 2, 1992. 

5 In operation, each of the four-axis devices 

136 generate a series of signals 139 representative of 
the position and orientation of the corresponding 
instrument 134. These signals are transmitted to a 
controller 138 which generates a series of force com- 
10 mands 137 which are returned to the four-axis devices 
136 to provide force feedback on the corresponding 
instrument 134 . 

The controller 138 may be a virtual reality 
simulation of the environment of a medical procedure, 

15 formed either artificially or based upon data collected 
during an actual medical procedure. The controller 138 
could also represent one or more slave devices which 
respond to follow the actions of the instruments 134, 
measure the forces acting on the slave device, and 

20 transmit force commands 137 back to the four-axis 
devices 136. 

It should be noted that there are a variety of 
different ways that the four-axis device could be 
implemented. This device could read the position of a 
25 user's hand by means of a laser tracking system and 
feedback forces by urging the user's hand with jets of 
air. In the preferred embodiment, however, the user's 
hand will grip a member 134 representative of a medical 
instrument or another medical tool. 
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The specific four-axis device shown in Figure 
10 is now examined in more detail. The position, 
orientation, velocity, and/or acceleration provided on 
the force reflective mechanism 140 by a member 134 is 
5 sensed and transmitted as a command to the simulation 
which implements a virtual reality force field. In 
turn, a force command is generated based on the force 
field value for the position, orientation, velocity, 
and/or acceleration of the member 134 given by the 
10 simulation. This force command is transmitted to force 
reflective mechanism 140. Force is then generated on 

■ 

the member 134 by the force reflective mechanism 140 
using four small, brushless, DC servo motors. The four 
axis force-reflection output and four-axis orientation 
15 and position control provides the user with direct 
kinesthetic feedback from the virtual environment 
traversed. A queuing button (not shown) can be incorpo- 
rated in order to actuate the instrument within the 
simulation or control. 

20 The invention in its preferred embodiment is 

applicable to controlling a virtual or simulated medical 
environment. However, the invention is also well suited 
to the control of a remote or physical medical instru- 
ment. Further, the present invention is suited for 

25 application to any number of axes. For instance, a full 
six-axis interface could be used for simulating or 
controlling medical procedures. 

While the best mode for carrying out the 
invention has been described in detail, those familiar 
30 with the art to which this invention relates will 
recognize various alternative designs and embodiments 
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for practicing the invention as defined by the following 
claims. 
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What Is Claimed Is : 

• 

1. A method for simulating actual medical 
procedures, the actual medical procedure utilizing at 
least one medical instrument, the method comprising the 

5 steps of: 

measuring the actions of the medical instru- 
ment during an actual medical procedure; and 

generating the actions on a member representa- 
tive of the medical instrument; 
10 whereby generating the actions simulates the 

actual medical procedure. 

2. The method of claim 1 further comprising 
the step of recording the measured actions. 

3- The method of claim 2 wherein the step of 
15 generating the actions on the member is performed based 
upon the recorded actions. 

4. The method of claim 1 wherein the actual 
medical procedure includes endoscopic surgery utilizing 
a viewing scope capable of generating images. 

20 5. The method of claim 4 further comprising 

the step of displaying the generated images. 

6 . The method of claim 5 further comprising 
the step of recording the images generated by the 
viewing scope wherein the step of displaying is per- 

25 formed based upon the generated images as recorded. 

7. The method of claim 4 wherein the actual 
medical procedure includes preoperative exploration with 
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4 

the viewing scope and wherein the medical instrument 
comprises the viewing scope . 

8. The method of claim 5 further comprising 
the step of measuring supplemental information, wherein 

5 the step of displaying is performed in conjunction with 
the playback of the supplemental information. 

9. The method of claim 8 wherein the supple- 
mental information is audio information. 

10 10. The method of claim 8 wherein the supple- 

mental information is derived from medical diagnostic 
equipment . 

11. The method of claim 6 wherein the step of 
displaying is performed based upon diagnostic informa- 

15 tion. 

12. A system for simulating actual medical 
procedures, the actual medical procedure utilizing at 
least one medical instrument, the system comprising; 

20 measuring means for measuring the actions of 

the medical instrument during an actual medical proce- 
dure ; and 

generating means operatively associated with 
the measuring means for generating the actions on a 
25 member representative of the medical instrument; 

whereby the actions of the member simulate the 
actual medical procedure. 

13 . The system of claim 12 further comprising 
recording means operatively associated with the measur- 

30 ing means for recording the measured actions. 
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14. The system of claim 13 wherein the 
generating means is operatively associated with the 
recording means and wherein the generation of the 
actions on the member is performed based upon the 

5 recorded actions. 

15. The system of claim 12 wherein the actual 
medical procedure includes endoscopic surgery utilizing 
a viewing scope capable of generating images. 

16. The system of claim 15 further comprising 
10 display means operatively associated with the viewing 

scope for displaying the images generated by the viewing 
scope . 

17. The system of claim 16 further comprising 
recording means operatively associated with the viewing 

15 scope and the display means for recording the generated 
images wherein the display of the generated images is 
performed based upon the recorded images. 

18. The system of claim 15 wherein the actual 
medical procedure includes preoperative exploration with 

20 the viewing scope and wherein the medical instrument 
comprises the viewing scope. 

19. The system of claim 16 further comprising 
supplemental information means, operatively associated 
with the display means, for measuring supplemental 

25 information, wherein the display of the generated images 
is performed in conjunction with the playback of the 
supplemental information. 
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20* The system of claim 19 wherein the 
supplemental information means measures audio informa- 
tion. 

5 21. The system of claim 20 wherein the 

supplemental information means measures information from 
medical diagnostic equipment. 

22. The system of claim 18 wherein the 
display means further comprise means for displaying 

10 diagnostic information. 

23. A method for providing a virtual reality 
in response to a position and orientation of a member 
representative of a medical instrument, the method 

4 

15 comprising the steps of: 

generating an electrical signal for each of a 
plurality of degrees of freedom of the member as a 
function of the position and orientation of the member 
in three-dimensional space; 
20 generating at least one virtual reality force 

field in response to the generated electric signals; 

generating a force signal for each degree of 
freedom as a function of the force field; and 

directing a force on the member for each force 
25 signal, the generated forces providing the virtual 
reality. 

24. The method of claim 23 wherein the 
virtual reality force field is also generated in re- 
sponse to information obtained from an actual medical 

30 procedure. 
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25. The method of claim 23 wherein the 
virtual reality force field is also generated in re- 
sponse to information obtained during pre -medical 
diagnostic procedures. 

5 26. The method of claim 23 wherein the gener- 

ated forces are tactile forces. 

27. A method for controlling a medical 
instrument during a medical procedure, the method 
comprising the steps of: 

10 measuring the position and orientation of a 

member representative of the medical instrument; 

generating actions on the medical instrument 
based upon the measured position and orientation of the 
member; 

15 measuring the forces acting upon the medical 

instrument; and 

generating forces on the member based upon the 
measured forces on the medical instrument. 

28. The method of claim 27 wherein the step 
20 of measuring the forces acting upon the medical instru- 
ment includes measuring torques acting upon the medical 
instrument . 

29. The method of claim 28 wherein the step 
of generating the forces on the member includes generat- 

25 ing torques on the member. 

30. The method of claim 27 wherein the 
medical instrument includes an endoscopic medical 
viewing scope capable of generating images. 



WO 95/02233 



PCT/US94/07218 



-24- 

31. The method of claim 30 further comprising 
the step of displaying the generated images. 

32. The method of claim 30 wherein the 
medical procedure includes preoperative exploration with 

5 the viewing scope. 

33. The method of claim 31 further comprising 
the step of measuring supplemental information, wherein 
the step of displaying is performed in conjunction with 
the playback of the supplemental information. 

10 34. The method of claim 33 wherein the 

supplemental information is audio information. 

35. The method of claim 33 wherein the 
supplemental information is derived from medical diag- 

15 nostic equipment. 

36. The method of claim 33 wherein the step 
of displaying is performed based upon diagnostic infor- 
mation. 

37. The method of claim 27 further comprising 
20 the step of actuating the medical instrument in response 

* 

to a command input. 

38. A system for controlling a medical 
instrument during a medical procedure, the system 
comprising: 

25 first measuring means for measuring the 

position and orientation of a member representative of 
the medical instrument; 
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first generating means operatively associated 
with the first measuring means for generating actions on 
the medical instrument based upon the measured position 
and orientation of the member; 
5 second measuring means for measuring the 

forces acting upon the medical instrument; and 

second generating means operatively associated 
with the second measuring means for generating forces on 
the member based upon the measured forces on the medical 
10 instrument, 

39. The system of claim 38 wherein the second 
measuring means includes means for measuring torques 
acting upon the medical instrument. 

40. The system of claim 39 wherein the second 
15 generating means includes means for generating torques 

on the member. 

41. The system of claim 38 wherein the 
medical instrument includes an endoscopic medical 
viewing scope capable of generating images. 

20 42. The system of claim 41 further comprising 

display means operatively associated with the viewing 
scope for displaying the generated images. 

43. The system of claim 41 wherein the 
medical procedure includes preoperative exploration with 

25 the viewing scope. 

44. The system of claim 41 further comprising 
supplemental information means, operatively associated 
with the display means, for measuring supplemental 
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information, wherein the display of the generated images 
is performed in conjunction with the playback of the 
supplemental information. 

45. The system of claim 44 wherein the 
5 supplemental information means measures audio informa- 
tion. 

46. The system of claim 45 wherein the 
supplemental information means measures information from 

10 medical diagnostic equipment. 

47. The system of claim 43 wherein the 
display means further comprise means for displaying 
diagnostic information. 

48. The system of claim 38 further comprising 
15 actuation means operatively associated with the medical 

instrument for actuating the medical instrument in 
response to a command input. 

49. A system for providing four-axis control 
for a medical instrument comprising: 

20 a member representative of the medical instru- 

ment ; 

a four-axis device securable to the member, 
the four-axis device being responsive to the position 
and orientation of the member to generate electrical 
25 signals representative of the position and orientation 
of the member, the four-axis device including an actua- 
tor for each axis of the device; 

means for actuating the medical instrument in 
response to the electrical signals; 
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means for generating a force field based upon 
a force induced upon the medical instrument; and 

means for generating a force signal for each 
axis of the four-axis device as a function of the force 
5 field, wherein the actuators are responsive to their 
respective force signals to provide a force to each axis 
of the four-axis device for reflecting the force on the 
medical instrument to the member; 

whereby the position and orientation of the 
10 member control the medical instrument. 

50. The system of claim 49 wherein the four- 
axis device further comprises an encoder coupled to each 
actuator for generating the corresponding electrical 
signals. 

15 51. The system of claim 49 wherein the 

actuators are electric motors. 

52. The system of claim 51 wherein the 
electric motors are back-drivable. 
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